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ABSTRACT 
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•  r 

The  Multi-Sensor  Feature  Level  Fusion  (MSFLF)  program  is  intended  to  improve  automatic  target 
recognition  (ATR)  performance  by  the  use  of  multi-sensor  feature  techniques.  2WSFLF  algorithms  are 
evaluated  using  multi-sensor  (8-12m  FLIR,  94GHz  MMW  radar  and  10.6m  CO2  laser  radar)  data 
and  imagery.  This  report  gives  the  results  of  target  detection  and  identification  using  the  MSFLF 
training  IR  imagery.  A  set  of  image  quality  measures  are  defined  and  computed  over  the  imagery. 
These  quality  measures  suggest  that  performance  should  be  better  for  the  training  imagery  than  for 
the  test  imagery.  An  overview  of  the  FLIR  and  MMW  radar  algorithms,  detailed  descriptions  of  the 
IR  target  detection  and  identification  algorithms  are  presented  in  a  seperate  proprietary  report. 
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1.0  Introduction 

The  FLIR/MMW  radar  target  identification  algorithms  have  been  applied  to  three  different  data 
sets;  Phase  1,  Vision  1  Training,  and  Vision  1  Testing.  The  Phase  1  and  Vision  1  Training  FLIR 
imagery  was  used  with  simulated  MMW  radar  range  data.  The  limited  amount  of  multi-sensor 
data  precluded  true  FLIR/MMW  radar  sensor  fusion  target  identification  for  these  data  sets.  These 
FLIR  data  sets  were  utilized  in  the  development  of  the  target  detection  and  identification  algorithms. 
Target  identification  results  using  the  Phase  1  and  Vision  1  Training  FLIR  imagery  are  presented  in 
this  report.  The  Vision  1  Test  data  contains  both  FLIR  imagery  and  MMW  radar  data.  The  target 
detection  and  identification  performance  for  the  Vsion  1  Test  dataset  is  not  presented  in  this  report 
since  it  was  scored  by  the  government 

Target  identification  has  been  performed  at  two  different  levels.  One  level  performs  identification 
given  ideal  locations  of  the  target.  The  true  targets  are  ground  truthed  with  boxes  placed  on  the  FLIR 
images  around  the  targets  during  a  manual  operation.  This  ideal  detection  gives  an  indication  of  the 
best  performance  of  the  identification  process.  The  second  level  uses  the  detection  algorithm  to 
detect  and  place  boxes  around  the  estimated  target  locations.  This  method  results  in  missed  targets, 
false  alarms,  and  partly  miss-placed  boxes. 

The  identification  process  identifies  the  target  as  a  specific  type,  e.g.  Ml  15  APC.  The  recognition 
process  is  a  grouping  of  targets  into  three  groups;  tank,  APC,  and  truck.  The  classification  process  is 
also  grouping  of  targets  but  only  into  two  classes;  tracked  and  wheeled.  Unlike  the  single  sensor  cases 
were  classification  and  recognition  are  performed,  the  FLIR/MMW  radar  algorithm  performs  target 
identification  only.  Recognition  and  classification  are  a  grouping  of  the  results  of  the  identification 
process. 

The  characteristics  of  the  FLIR  imagery  are  critical  to  target  detection  and  identification  perfor¬ 
mance.  Image  quality  measures  (QM)  have  been  defined  which  measure  the  clutter  and  target  signal 
level  in  the  region  around  the  target.  They  use  the  target’s  pixel  location  box  defined  in  the  ground 
truthing  of  the  FLIR  imagery.  The  QM’s  are  intended  to  estimate  expected  detection  and  identifica¬ 
tion  algorithm  performance.  The  statistics  of  these  QM’s  provide  relative  target-to-clutter  between 
different  data  sets  and  are  correlated  to  subsequent  target  detection  and  identification  performance. 


2.0  Target  Detection  and  Identification  Results 

2.1  Phase  1  FLIR  Imagery  Results 

The  Phase  1  FLIR  imagery  field-of-view  (fov)  is  3.43®  x  2.58°  for  the  narrow  fov  and  10.32°  x 
7.74°  for  the  wide  fov.  The  target  range  varies  from  500m  to  1700m,  most  are  near  1000m,  with  up 
to  four  targets  in  the  wide  fov.  The  target  location  varied  from  in  the  open  with  little  clutter,  to  along 
the  tree  line  with  considerable  clutter.  The  Phase  1  FLIR  imagery  contains  only  five  target  types: 
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1.  M151  Jeep 

2.  M35  Truck 

3.  Ml  13  APC 

4.  M60  Tank 

5.  M551  Tank 

The  M60  tank  is  represented  by  both  the  long  barrel  M60A1  and  the  short  barrel  M60A2.  Two 
different  trucks  are  present  in  the  Phase  1  imagery  but  a  single  M35  model  was  used. 

Table  2.1  presents  the  results  of  target  detection  and  identification  applied  to  Phase  1  FLIR 
imagery,  both  narrow  and  wide  fov  imagery.  The  results  presented  for  Phase  1  are  for  the  older 
version  of  the  spoke  filter  detection  algorithm.  The  recognition  and  identification  percentages  are 
conditioned  on  the  occurance  of  target  detection.  Since  five  target  models  are  used  the  expected 
random  identification  is  20%.  The  expected  random  target  recognition  is  33%  since  their  are  three 
recognition  groups;  tank,  APC  and  truck. 


Table  2.1  Phase  1  Imagery  Target  Detection,  False  Alarm,  Recognition,  and  Identification  Performance 


FOV 

AREA 

IMAGES 

TARGETS 

DETECTIONS 

FALSE  ALARMS 

REC 

ID 

sqdeg 

/image 

/sq  deg 

cond 

cond 

NARROW 

8.85 

449 

834 

497 

60% 

114 

0.25 

58% 

47% 

WIDE 

79.88 

319 

659 

276 

42% 

12 

0.04 

42% 

31% 

1  Expected  random  results 

33% 

20% 

The  identification  performance  was  evaluated  using  Phase  1  FLIR  imagery  under  ideal  detection. 
This  allows  for  investigation  of  the  identification  performance  of  all  the  targets  present  in  the  imagery, 
not  just  those  detected  by  the  specific  detection  algorithm.  Using  ideal  detection  boxes  the  results 
are  given  by  Table  2.2.  It  is  clear  that  the  identification  increases  by  9  and  5%  points  for  the  narrow 
and  wide  fov  cases  using  ideal  detection.  This  is  even  more  significant  since  the  additional  targets 
(825  vs  497  narrow  fov)  which  were  not  detected  by  the  detection  algorithm  are  in  general  of  poorer 
quality,  and  are  included  for  identification  under  the  ideal  detection  case  of  Table  2.2. 

Table  2.2  Phase  1  Thrget  Recognition  and  Identification  Performance  with  Ideal  Detection 


FOV 

IMAGES 

TARGETS 

REC 

ID 

NARROW 

442 

825 

63% 

56% 

WIDE 

450 

837 

49% 

36% 

1  Expected  random  results 

33% 

20% 

The  results  presented  in  Tables  2. 1  and  2.2  were  computed  at  different  times  consequently  they 
do  not  contain  exactly  the  same  images.  Seven  of  the  narrow  fov  images  were  discarded  due  to 
extreme  poor  quality.  One  hundred  thirty-one  additional  wide  fov  images  were  available  when  the 
ideal  identification  performance  of  Table  2.2  was  computed. 
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In  order  to  further  investigate  the  identification  performance  the  Phase  1  narrow  fov  results  are 
divided  into  different  groups  of  data.  Table  2.3  presents  these  results  for  the  specific  target  types  of 
the  different  groups  of  imagery.  Some  of  the  imagery  is  high  clutter  while  the  others  have  targets 
which  partially  obscure  other  targets.  The  last  group  has  a  few  camouflaged  M60  tanks. 


Table  2.3  Phase  1  Target  Recognition  and  Identification  Perfwmance  by  Groups  with  Ideal  Detection  G'larrow  FOV) 


IMAGE 

COMMENTS 

TARGETS 

IDENTIFICATION 

TOTAL  1 

FILES 

TANK 

APC 

TRUCK 

JEEP 

TANK 

MSS1 

Ml  13 

Ml  51 

REC 

REC 

ID 

note  A 

clutter 

141 

82% 

- 

69% 

- 

0 

92% 

84% 

77% 

dd0271,3 

tree  line 

77 

77% 

54% 

- 

0 

0 

90% 

70% 

56% 

df0871 

SOOm  rang* 

26 

56% 

- 

- 

90% 

- 

62% 

73% 

69% 

df1271 

3  tgt/lmag* 

81 

59% 

- 

52% 

56% 

- 

85% 

64% 

56% 

df1572 

P.O. 

127 

- 

20% 

74% 

- 

9% 

42% 

44% 

36% 

note  B 

low  contrast 

278 

50% 

19% 

76% 

36% 

- 

53% 

50% 

47% 

df2371,2- 

P.O. 

86 

- 

- 

89% 

58% 

- 

- 

72% 

72% 

df2372-f 

camouflag* 

8 

63% 

- 

- 

- 

- 

63% 

63% 

63% 

TOTAL 

824 

74% 

23% 

74% 

47% 

6% 

73% 

63% 

56% 

TRUE  TARGETS 

824 

208 

123 

246 

194 

54 

331 

824 

824 

ID  AS  TYPE 

824 

278 

100 

302 

106 

26 

378 

- 

- 

CORRECT 

824 

154 

28 

182 

91 

3 

243 

519 

458 

P.O.  s  Some  of  the  targets  are  partially  obscured,  two  or  three  targets  per  Image, 
camouflage  s  One  of  the  two  M60  tank  targets  per  image  Is  camouflaged. 

Of  the  824  targets,  13  (2%)  were  incorrectly  identified  as  clutter, 
note  A  =  dc2571,dc2572,dc2672,dc2673,df0571,df1071 
note  B  =  df  1 971  ,df1972,df2271,df2272,df 2471 


As  is  evident  from  Table  2.3,  identification  performance  varies  from  36%  to  77%  (56%  average), 
while  recognition  varies  from  44%  to  84%  over  the  8  groups  of  imagery.  The  M151  jeep  was  the 
most  difficult  to  identify.  This  is  in  part  due  to  its  small  size  but  also  most  of  the  jeep  images 
are  in  the  high  clutter  environment.  The  best  identification  performance  (74%)  is  of  the  M60  Tank 
and  Ml  13  APC. 

The  correct  identification  and  miss-identification  between  target  types  is  illustrated  in  the  confu¬ 
sion  matrix  of  Table  2.4.  The  columns  of  the  matrix  list  the  number  of  targets  of  each  type  which 
are  identified  as  the  target  type  of  the  column  heading.  For  example,  278  targets  were  identified  as 
M60  tanks,  124  of  these  identifications  were  incorrect  (’BAD’)  and  154  were  correctly  identified. 
The  correct  identification  falls  along  the  matrix  diagonal.  Summing  across  each  row  gives  the  true 
number  of  each  target  type.  A  total  of  13  targets  are  incorrectly  identified  as  clutter  (GLUT). 


Final  Report  —  MSFLF 

Table  2.4  Phase  1  Target  ConfusicHi  Matrix  (Ideal  Detection,  Narrow  FOV) 


IDENTIFICATiON  TARGET  TYPE 

TANK 

APC 

TRUCK 

TRUE 

GLUT 

MS51 

M113 

M35 

Ml  51 

7 

154 

25 

11 

8 

3 

M551 

H 

36 

28 

57 

0 

M113 

D 

24 

21 

182 

14 

1 

11 

19 

32 

91 

6 

Ml  51 

5 

n 

D 

20 

3 

TOTAL 

13 

278 

100 

302 

106 

26 

BAD 

13 

124 

72 

120 

15 

23 

The  percentage  correct  identification  and  miss-identification  between  target  types  are  illustrated 
in  the  confusion  matrix  of  Table  2.5.  This  table  is  based  on  the  numbers  presented  in  Table  2.4. 
The  identification  numbers  are  divided  by  the  number  of  true  targets  of  each  type.  For  example 
reading  across  the  M60  row,  74%  of  the  true  M60  targets  are  correctly  identified,  with  the  most 
common  (12%)  miss  identification  being  the  M551  tank.  The  bottom  row  of  the  table  (’BAD’) 
presents  percentage  of  incorrect  identifications  for  the  specific  target  type  of  the  column  heading.  For 
example,  100  targets  are  identified  as  M551  of  these  28  are  correct,  giving  72%  ’BAD’  identification 
of  those  identified  as  an  M551  tank.  This  indicates  that  very  few  of  the  targets  are  incorrectly 
identified  as  the  M35  truck.  The  last  row  (’BIAS’)  indicates  the  percentage  of  targets  identified  as 
the  type  relative  to  the  true  number  of  that  type  of  target 

Table  2.S  Phase  1  Percentage  Target  Confusion  Matrix  (Ideal  Detection,  Narrow  FOV) 


IDENTIFICATION  TARGET  TYPE 

TANK 

APC 

TRUCK 

TRUE 

GLUT 

M551 

M113 

M35 

Ml  51 

3% 

74% 

12% 

5% 

4% 

1% 

M551 

n 

29% 

23% 

2% 

0 

M113 

B 

10% 

9% 

n 

2% 

6% 

M35 

1% 

23% 

10% 

16% 

47% 

3% 

Ml  51 

9% 

35% 

13% 

31% 

0 

6% 

BAD 

100% 

45% 

72% 

40% 

14% 

88% 

BIAS 

- 

134% 

81% 

123% 

55% 

48% 
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The  following  lists  some  of  the  identification  relationships  between  the  different  target  types. 


1.  The  M60  tank  and  Ml  13  APC  targets  are  identified  correctly  74%  of  the  time  with  little 
confusion  (<12%)  with  other  target  types. 

2.  Of  the  targets  identified  as  M60  tanks  55%  are  correct 

3.  Of  the  targets  identified  as  Ml  13  APCs  60%  are  correct. 

4.  The  identification  bias  is  toward  the  M60  tank  and  Ml  13  APC,  with  the  M35  truck  and 
M151  jeep  least  likely. 

5.  While  the  M35  truck  and  the  M151  jeep  are  correctly  identified  only  47%  and  6%  of  the 
time,  other  targets  are  seldom  miss-identified  as  either. 

6.  Of  the  targets  identified  as  M151  jeeps  only  12%  are  correct. 

7.  Of  the  targets  identified  as  M35  trucks  86%  are  correct 

8.  The  Ml 5 1  jeep  and  M55 1  tank  are  the  only  target  types  which  are  less  frequently  identified 
as  the  correct  target  type  versus  some  other  target  type.  Both  the  M551  tank  and  M151 
jeep  are  more  frequently  identified  as  the  Ml  13  APC. 

The  wide  fov  confusion  matrix  is  presented  in  Table  2.6.  The  M60  tank  is  the  most  common 
target  (217  occurrences)  but  even  more  targets  were  identified  as  M60’s  (366)  of  these  212  are 
incorrect.  The  least  common  target  is  the  M 151  jeep  (53  occurrences)  but  only  16  of  the  targets  are 
identified  as  M15rs.  One  M35  target  is  incorrectly  identified  as  clutter  (CLUT). 


Table  2.6  Phase  1  Target  Confusion  Matrix  (Ideal  Detection,  Wide  FOV) 


IDENTIFICATION  TARGET  TYPE 

TANK 

APC 

TRUCK 

TRUE 

CLUT 

M60 

M551 

M113 

M35 

Ml  51 

n 

154 

24 

11 

23 

5 

M551 

D 

49 

18 

12 

8 

2 

M113 

0 

71 

33 

14 

1 

M35 

II 

71 

36 

li 

24 

0 

Ml  51 

D 

21 

8 

H 

6 

8 

TOTAL 

1 

366 

119 

81 

75 

16 

TRUE 

a 

217 

89 

152 

147 

53 

BAD 

1 

212 

101 

48 

51 

8 

1 
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2.2  Vision  1  TVaining  FLIR  Imagery  Results 

The  Msion  1  training  imagery  consists  of  FLIR  images  of  a  single  target  at  a  range  of  about 
lOOOkm.  The  imagery  is  taken  with  the  target  rotated  at  5°  increments.  It  is  generally  a  low  clutter 
environment  with  most  of  the  differences  in  image  quality  resulting  from  the  weather  conditions, 
vehicle  operation  and  time  of  day.  The  initial  Vision  1  data  contains  the  same  targets  as  Phase  1 
data,  with  the  exception  of  the  M151  jeep.  Later  Vision  1  data  included  additional  target  types,  in 
total  nine  targets  are  included. 

1.  M35  Truck 

2.  Ml  13  APC 

3.  M60  Main  Battle  Tank,  both  M60A1  and  M60A2 

4.  M551  Sheridan  Light  Tank 

5.  Ml  Abrams  Main  Battle  Tank 

6.  M2  Bradley  Infantry  Fighting  Vehicle 

7.  MHO  Self-propelled  Howitzer 

8.  Ml 63  Vulcan  Air  Defense  System 

9.  M578  Light  Armored  Recovery  Vehicle 

Table  2.7  presents  the  results  of  target  detection  and  identification  over  the  \fision  1  Training 
imagery,  both  narrow  and  wide  fov.  The  spoke  filter  algorithm  which  performs  detection  has  been 
modified  since  these  results  were  computed.  The  results  presented  for  Vision  1  Training  are  for 
the  older  version  of  the  detection  algorithm.  The  recognition  and  identification  percentages  are 
conditional  given  that  a  detection  occurs.  Since  ten  target  models  are  used,  including  the  two 
variations  of  the  M60,  the  expected  random  identification  is  10%.  The  set  A  data  consists  of  the 
initial  five  targets  (M60A1,  M60A2,  M551,  Ml  13,  and  M35).  The  set  B  data  consists  of  the  five 
targets;  Ml,  M2,  MHO,  M163,  and  M578.  The  initial  set  of  imagery  (set  A)  was  ran  using  five 
target  models  while  the  second  set  of  imagery  (set  B)  was  ran  using  the  total  of  ten  target  models. 
The  combined  set  of  imagery  (sets  A&B)  was  later  ran  with  the  total  set  of  ten  target  models.  As 
should  be  expected  the  results  are  better  when  only  five  target  models  are  required.  The  wide  fov 
set  B  was  not  ran  since  narrow  fov  was  of  primary  interest. 


Table  2.7  Vision  1  Training  Target  Detection,  False  Alarm,  Recognition,  and  Identification  Performance 


FOV 

SET 

AREA 

MODELS 

TGTS 

FALSE  ALARMS 

REG 

IDENTIFICATION 

/jq  deg 

cond 

ccnditionil 

NARROW 

A 

3.35 

5 

210 

96% 

68 

0.32 

0.18 

68% 

63% 

NARROW 

B 

2.61 

10 

314 

n 

90% 

35 

0.11 

0.07 

51% 

19% 

NARROW 

A&B 

2.91 

10 

524 

482 

92% 

103 

0.20 

0.09 

58% 

37% 

WIDE 

29.63 

5 

211 

160 

76% 

20 

0.09 

0.04 

44% 

41% 

1  Expected  random  results 

33% 

20%  10% 

Set  A  contains  inital  five  targets  (M60A1,  M60A2,  M551,  M113,  M35). 

Set  B  contains  final  five  targets  (Ml,  M2,  MHO,  M163,  and  M578). 
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The  identification  performance  was  evaluated  using  Vision  1  Training  FLIR  imagery  under  ideal 
detection.  This  allows  for  investigation  of  the  identification  performance  of  all  the  targets  present 
in  the  imagery,  not  just  those  detected  by  the  specific  detection  algorithm.  Using  ideal  detection 
boxes  the  results  are  given  by  Table  2.8.  Identification  using  ideal  detection  increases  by  1%  points 
(37%  to  44%)  for  narrow  fov  with  ten  target  models  (sets  A&B)  but  decreases  12%  points  (41% 
to  29%)  for  the  wide  fov  case.  Performance  was  improved  using  the  total  imagery  set  (A«S:B)  by 
minor  modification  of  the  target  models  and  reduction  in  the  number  of  sensed  image  edge  pixels 
considered.  The  identification  improved  from  35%  to  44%. 


Table  2.8  Vision  1  Target  Recognition  and  Identification  Perfonnance  with  Ideal  Detection 


FOV 

SET 

MODELS 

TARGETS 

REC 

ID 

NARROW 

A 

5 

210 

71% 

67% 

NARROW 

B 

5 

314 

66% 

47% 

NARROW 

A 

10 

55% 

43% 

NARROW 

B 

10 

■■ 

56% 

32% 

NARROW 

A&B 

10 

524 

55% 

35% 

1  Improved  performance  (A&B,  10  Models)* 

63% 

44% 

A 

5 

211 

32% 

29% 

B 

10 

318 

44% 

14% 

I  Expected  random  results 

33% 

20%  10% 

I  *  Performance  was  improved  by  modification  of  the  target  models  and  reduction  in  the  number  of  edge  pixels.  | 

The  target  identification  performance  for  each  of  the  nine  target  types,  the  M60A1  and  A2  are 
grouped  together,  using  the  ideal  detections  is  presented  in  Table  2.9.  There  are  a  total  of  524  targets 
of  which  (229)  44%  are  correctly  identified.  This  44%  correct  identification  is  the  average  over  all 
the  different  target  types.  The  poorest  identification  (10%)  is  for  the  M551  tank,  while  the  best  (84%) 
is  for  the  Ml  13  APC.  The  three  groups  used  for  target  recognition  are  somewhat  arbitrary. 
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l^ble  2.9  Vision  1  Target  Type  Identification  using  Ideal  Detection 


TANK  CLASS 

APC  CLASS 

TRUCK 

M551 

M60 

Ml  M2 

Ml  10 

M578 

M113  M163 

M35 

NUMBER  IDENTIFIED  AS 
THIS  TYPE 

26 

75 

55  80 

33 

29 

112 

55 

59 

NUMBER  TRUE  TARGETS 

OF  THIS  TYPE 

10 

63 

69  115 

72 

7 

85 

51 

52 

NUMBER  CORRECT 
IDENTIFICATION 

1 

29 

26  46 

14 

4 

71 

14 

24 

%  CORRECT 
IDENTIFICATION 

10% 

46% 

38%  40% 

19% 

57% 

84% 

27% 

46% 

NUMBER  TRUE  TARGETS 

OF  THIS  CLASS 

336 

136 

52 

NUMBER  CORRECT 
RECOGNITION 

254 

103 

24 

%  CORRECT 

RECOGNITION 

76% 

76% 

46% 

The  correct  identification  and  miss-identification  between  target  types  is  illustrated  in  the  confu¬ 
sion  matrix  of  Table  2.10.  The  columns  of  the  matrix  list  the  number  of  targets  of  each  type  which  are 
identified  as  the  target  type  of  the  column  heading.  For  example,  75  targets  were  identified  as  M60 
tanks,  46  of  these  identifications  were  incorrect  (’BAD’)  and  29  were  correctly  identified.  The  largest 
number  of  the  incorrectly  identified  as  M60  tanks  (19)  are  in  truth  Ml  lO’s.  The  correct  identification 
falls  along  the  matrix  diagonal.  Summing  across  each  row  gives  the  true  number  of  each  target  type. 
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Table  2.10  Vision  I  Target  Confusion  Matrix  (Ideal  Detection,  Narrow  FOV) 


IDENTIFICATION  TARGET  TYPE 

TANK  CLASS 

APC  CLASS 

TRUCK 

TRUE 

M551 

M60 

Ml 

M2 

M110 

M578 

M113 

M163 

M35 

M551 

1 

1 

3 

2 

0 

0 

0 

1 

2 

M60 

2 

29 

3 

4 

7 

1 

2 

10 

5 

Ml 

2 

8 

26 

6 

3 

2 

6 

8 

8 

M2 

10 

7 

15 

46 

8 

10 

5 

4 

M110 

6 

19 

6 

6 

14 

2 

8 

5 

6 

M578 

0 

0 

0 

1 

0 

4 

1 

1 

0 

M113 

2 

3 

1 

0 

0 

2 

D 

3 

M163 

3 

6 

1 

9 

0 

2 

1 

M35 

2 

0 

6 

1 

6 

4 

9 

24 

TOTAL 

26 

75 

55 

80 

33 

29 

112 

55 

59 

BAD 

25 

46 

29 

34 

19 

25 

41 

41 

35 

The  percentage  correct  identification  and  miss-identification  between  target  types  is  illustrated  in 
the  confusion  matrix  of  Table  2.11.  This  table  is  based  on  the  numbers  presented  in  Table  2.10.  The 
identification  numbers  are  divided  by  the  number  of  true  targets  of  each  type.  For  example  reading 
across  the  M60  row,  46%  of  the  true  M60  targets  are  correctly  identified,  with  the  most  common 
(16%)  miss-identification  being  the  M163.  The  bottom  row  of  the  table  (’BAD’)  presents  percentage 
of  incorrect  Mentifications  for  the  specific  target  type  of  the  column  heading.  For  example,  26  targets 
are  identified  as  M551  of  these  only  1  is  correct,  giving  96%  ’BAD’  identification  of  those  identified 
as  M551  tank. 
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Table  2.11  Vision  1  Percentage  Target  Confusion  Matrix  (Ideal  Detection,  Narrow  FOV) 


IDENTIFICATION  TARGET  TYPE 

TANK  CLASS 

_ 

APC  CLASS 

TRUCK 

TRUE 

M551 

M60 

Ml 

M2 

MHO 

M578 

M113 

Ml  63 

M35 

M551 

10% 

10% 

30% 

20% 

0 

0 

0 

10% 

20% 

M60 

3% 

40% 

5% 

6% 

11% 

2% 

3% 

16% 

8% 

Ml 

3% 

12% 

3S% 

9% 

4% 

3% 

9% 

12% 

12% 

M2 

9% 

6% 

13% 

40% 

7% 

9% 

4% 

3% 

9% 

M110 

8% 

26% 

8% 

8% 

19% 

3% 

11% 

7% 

8% 

M578 

0 

0 

0 

14% 

0 

57% 

14% 

14% 

0 

M113 

2% 

4% 

1% 

0 

0 

2% 

84% 

4% 

4% 

M163 

6% 

12% 

2% 

18% 

0 

4% 

29% 

27% 

2% 

M35 

0 

4% 

0 

12% 

2% 

12% 

8% 

17% 

46% 

BAD 

96% 

61% 

53% 

43% 

58% 

86% 

37% 

75% 

59% 

BIAS 

260% 

119% 

80% 

70% 

46% 

414% 

132% 

108% 

113% 

As  is  evident  from  the  confusion  matrices,  the  best  identification  performance  is  for  the  Ml  13 
APC,  with  84%  correct  identification  and  only  37%  miss — identified  as  the  Ml  13.  The  M163  is  an 
air  defense  gun  mounted  on  an  Ml  13  chassis,  consequently  29%  of  the  M163  are  miss-identified  as 
M113’s.  Also  the  M2  front  or  rear  view  is  very  much  like  the  180°  inverse  of  the  M163  resulting  in 
18%  miss-identification  of  the  M163  as  the  M2.  In  all  cases  except  for  the  M551  tank,  MHO  gun, 
and  Ml 63  r  defense  unit  the  correct  target  is  the  most  common  identification. 


3.0  FLIR  Image  Quality  Measures 

3.1  Definition  of  Quality  Measures 

Image  quality  measures  (QM)  estimate  the  clutter  and  target  signal  level  in  the  region  around  the 
target,  using  the  pixel  location  box  defined  in  the  ground  truthing  of  the  FLIR  imagery.  A  box  region 
expanded  by  one-half  of  the  ground  truthed  target  size  on  each  of  the  sides  around  the  target  is  used 
as  the  background.  The  ground  truthed  target  box  contains  the  region  used  for  the  target  statistics. 
The  target  excludes  a  border  of  2  pixels  which  are  not  counted  as  either  target  or  background.  The 
top  one-third  of  the  ground  truth  box  has  only  a  maximum  of  2m  at  the  center  used  as  the  target,  the 
rest  is  not  counted  as  target  or  background.  These  border  regions  are  defined  to  minimize  counting 
non-target  background  within  the  target  statistics. 
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Four  QM’s  are  defined.  The  first  three  QM’s  are  target- to- background  contrast  measures.  The 
fourth  QM  measures  the  difference  in  the  intensity  distributions  between  target  and  background.  The 
first  QM  is  the  difference  between  target  average  intensity  (Aj)  and  background  average  intensity 
(Ag)  normalized  by  the  image  intensity  standard  deviation  within  and  around  the  target  and  scaled 
by  the  square  root  of  the  number  of  target  pixels  (^/Nt). 

QM,  = 

Where  <7j  is  the  variance  of  intensity  within  the  target  region  and  <7^  is  the  variance  within  the 
background  region. 

The  second  QM  includes  the  difference  in  intensity  variation  within  the  target  and  background 
regions.  This  attempts  to  account  for  different  textures  between  target  and  background. 

QM2  = 

\/4  +  ^ 

The  third  QM  is  similar  to  the  first  except  order  statistics  are  used.  The  QM  is  based  upon 
intensity  percentile  locations  on  a  histogram  of  the  target  and  background  intensity  distributions. 

_ jPSOr  ~  _ 

^J(P80T  -  P20r)2  -f-  (P8O5  -  P20Bf 

Where  P20,  P50,  and  P80  are  the  20%,  50%  (median),  and  80%  intensity  levels  of  the  target  and 
background  regions. 

The  fourth  QM  is  based  upon  the  difference  between  the  histograms  of  the  target  and  background 
intensity  distributions.  The  QM  value  indicates  the  commonality  between  the  target  and  background 
IR  intensity  distributions. 


QM3  =  \/ N'T' 


QMA  =  -YMi 
n  f— ' 

1=1 

Where  M,  is  the  i‘*  peak  of  the  minimum  between  the  target  and  background  intensity  histograms, 
n  is  10  for  the  results  presented. 

3.2  FLIR  Image  Quality  Measure  Results 

The  QM’s  computed  over  the  narrow  fov  imagery  are  presented  in  Table  3.1.  The  first  three 
QM’s  increase  with  improved  image  quality,  since  they  are  contrast  measures.  The  Fourth  QM 
decreases  with  improved  image  quality,  since  it  measures  the  level  of  commonality  between  the 
target  and  background  regions.  The  Vision  1  Training  imagery  is  divided  into  sets  A  and  B  .  The 
A  set  corresponds  to  the  initial  set  of  five  targets  (M60A1,  M60a2,  M35,  Ml  13,  and  M551)  and 
the  B  set  corresponds  to  the  second  set  of  five  targets  (Ml,  M2,  M163,  M578,  and  MHO).  The 
Vision  1  Test  imagery  consists  of  three  images  which  cover  the  scene;  the  left,  center,  and  right  hand 
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images.  Each  of  the  QM’s  were  not  computed  over  all  of  the  data  due  to  time  limitations  and  limited 
computer  storage,  resulting  in  blanks  in  some  of  the  columns  of  Table  3.1. 

l^ble  3.1  FLIR  Imagery  Quality  Measures  Are  an  Indication  of  Expected  Detection  and  Identification  Perfonnance. 


DATASETS 

NUMBER 

OF 

TARGETS 

CONTRAST  MEASURES 

COMMONAUTY 

MEASURE 

OM1 

OM2 

QM3 

OM4 

mean 

aigma 

algma 

mean 

algma 

algma 

PHASE  1 

838 

16.7 

8.3 

VISION  1  TRAIN  A 

210 

13.9 

7.6 

9.2 

7.5 

5.8 

4.0 

VISION  1  TRAIN  B 

314 

17.9 

6.6 

10.3 

5.0 

6.3 

3.1 

VISION  1  TRAIN  A  &  B 

524 

16.3 

7.0 

9.9 

6.0 

6.1 

4.2 

VISION  1  TEST  LEFT 

104 

14.0 

6.4 

20.4 

6.0 

VISION  1  TEST  CENTER 

152 

8.4 

5.0 

14.2 

5.9 

VISION  1  TEST  RIGHT 

145 

8.1 

4.7 

12.7 

5.7 

VISION  1  TEST 

401 

12.0 

7.2 

15.2 

■a 

5.5 

It  is  evident  that  on  average  the  Phase  1  and  Vision  1  Training  imagery  target  contrast  is  better 
than  for  the  Vision  I  Test  imagery.  The  left  side  of  the  scene  for  the  Vision  1  Test  has  a  better  target- 
background  contrast  than  does  the  center  or  right  side.  This  is  easily  verified  by  visual  examination 
of  the  imagery.  The  center  and  right  side  have  greater  levels  of  bright  clutter  due  to  cleared  road 
surfaces. 
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